Enterprise organizations and the data analysts they employ face the challenge of finding useful information in the increasing amounts of data generated and collected by these organizations over time. Such “big data” may provide, for example, valuable insights into the organization's operational performance and business patterns associated with various parts of the organization. For example, accessing computer networks of a business enterprise and transmitting electronic communications across these networks generates massive amounts of data. Such data generated by machines may include, for example, Web logs of activity occurring at various web servers distributed across an enterprise's network.
Analysis of this data can indicate patterns in consumer behavior with respect to the particular products or brands in which consumers may be interested during a given period of time. Such pattern analysis also may be helpful in differentiating normal operational performance from anomalies. For example, the detection of unusual patterns can allow a system analyst to investigate the circumstances under which these unusual patterns emerged and determine whether any issues exist that may pose a threat to the system's operational performance or security. Moreover, analysis of such data allows business enterprises to understand how their employees, potential consumers, and/or Web visitors use the company's online resources. Such analysis can therefore provide businesses with operational intelligence, business intelligence, security intelligence, and a better overall ability to manage their information technology (IT) resources. For instance, such analysis may enable a business to better retain customers, meet customer needs, and improve the efficiency and security of the company's IT resources.
However, data analysts or systems administrators of an enterprise may encounter significant challenges when attempting to identify, collect, and analyze such large quantities of data, which may be distributed across multiple data sources within the enterprise's network environment or IT infrastructure. Such challenges may prevent these enterprise users from realizing the potential value that this data may provide. In particular, patterns in the enterprise's data as a whole, which may provide valuable insight into the operations of the enterprise, may be difficult to find due in part to the size of this data and the fact that the underlying data produced by each data source within the enterprise is usually analyzed in isolation, if at all.
The challenge of handling and analyzing large amounts of data may be particularly difficult in the context of operating a security management system. For example, a security information and event management (“SIEM”) system typically enables real-time analysis of security-related events generated based on computing activity across an enterprise. A SIEM system may also provide analytical tools having a range of functions including trend analysis, event identification, and alerting. Despite having an implemented SIEM system, many enterprises continue to battle a host of security vulnerabilities in their IT systems as distributed computing systems are rapidly adopted and expanded by the enterprises. With an expansion of distributed computing systems comes additional security issues due to the addition of the new components and the communications between these components. Such changes may introduce new challenges for monitoring and analyzing security events based on activity occurring in the distributed computing systems. Often, a user of a SIEM system may be presented with large amounts of data relating to security events occurring in the system. Left with a difficult task of sorting through the data to identify significant or noteworthy events, the user faces an additional challenge of indicating or flagging certain events to distinguish them from other events. Users also have difficulty aggregating or updating information used to identify significant or noteworthy security events.